Apparatus and method to train autonomous driving model, and autonomous driving apparatus

ABSTRACT

Disclosed is an apparatus and method to train an autonomous driving model. The apparatus includes a driver information collection processor configured to collect driver information while a vehicle is being driven. The apparatus also includes a sensor information collection processor configured to collect sensor information from a sensor installed in the vehicle while the vehicle is being driven, and a model training processor configured to train the autonomous driving model based on the driver information and the sensor information.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. Pat. No. 10,034,630, filed onNov. 15, 2016, which claims priority from Korean Patent Application No.10-2015-0160725, filed on Nov. 16, 2015, in the Korean IntellectualProperty Office, the disclosure of which is incorporated herein byreference in its entirety.

BACKGROUND 1. Field

The following description relates to an apparatus and a method to trainan autonomous driving model in an environment with real-drivingenvironment data and an autonomous driving apparatus and method usingthe autonomous driving model.

2. Description of Related Art

A general autonomous driving vehicle drives autonomously by recognizingand assessing an ambient environment using various sensors attached tothe vehicle. Such an autonomous driving vehicle recognizes an individualobject in the ambient environment using various sensors such as acamera, a light detection and ranging (lidar), and a radar installed forenvironment recognition during driving. In order to learn the recognizedindividual object, the autonomous driving vehicle performs a task oflabeling ground truth data of the object.

However, each of the sensors installed in the vehicle has a limitedrecognition range and low reliability problem. Accordingly, there arelimitations in autonomous driving technology that depends only onsensors installed in a vehicle. Furthermore, there are difficulties indefining autonomous driving rules for all surrounding conditions becausea real-road driving environment is very complicated.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

In accordance with an embodiment, there is provided an apparatus totrain an autonomous driving model, the apparatus may include: a driverinformation collection processor configured to collect driverinformation while a vehicle is being driven; a sensor informationcollection processor configured to collect sensor information from asensor installed in the vehicle while the vehicle is being driven; and amodel training processor configured to train the autonomous drivingmodel based on the driver information and the sensor information.

The vehicle is driven in at least one of a real-road driving situationand a virtual driving situation.

The driver information may also include at least one of driver stateinformation, driver behavior information, and driving patterninformation.

The driver state information may also include at least one ofinformation about a gaze direction, a heart rate, an electrocardiogram,and brain waves of a driver, and grip strength of the driver's handholding a steering wheel. The driver behavior information may alsoinclude at least one of information about manipulation of a stereodevice, air conditioning device, a mirror device, and a navigationdevice installed in the vehicle, and the driving pattern information mayalso include at least one of information about a steering wheelmanipulation information, horn manipulation information, turn-signalmanipulation information, emergency button manipulation information, andbrake pedal manipulation information.

The model training processor may train the autonomous driving model tolearn about a driving area based on information about the gaze directionof the driver and trains the autonomous driving model to learn at leastone of a dangerous situation and a level of danger in driving based onthe driving pattern information.

The apparatus may also include: a navigation information collectionprocessor configured to collect navigation setting information mayinclude information on a starting location and an ending location.

The model training processor may train the autonomous driving model tolearn about an integrated driving situation from the starting locationto the ending location and based on the navigation setting information.

The sensor information may also include at least one of surroundingimage information, surrounding voice information, current locationinformation, light detection and ranging (lidar) sensor information,radar sensor information, vehicle speed information, and accelerationinformation.

The autonomous driving model may be based on a deep neural networkconfigured to learn spatial information and temporal information.

In accordance with an embodiment, there is provided a method to train anautonomous driving model, the method may include: collecting driverinformation while a vehicle is being driven; collecting sensorinformation from a sensor installed in the vehicle while the vehicle isbeing driven; and training the autonomous driving model based on thedriver information and the sensor information.

The driver information may also include at least one of driver stateinformation, driver behavior information, and driving patterninformation.

The driver state information may also include at least one ofinformation about a gaze direction, a heart rate, an electrocardiogram,and brain waves of a driver, and grip strength of the driver's handholding a steering wheel.

The driver behavior information may also include at least one ofinformation about manipulation of a stereo device, air conditioningdevice, a mirror device, and a navigation device installed in thevehicle, and the driving pattern information may also include at leastone of a steering wheel manipulation information, horn manipulationinformation, turn-signal manipulation information, emergency buttonmanipulation information, and brake pedal manipulation information.

The training of the model may also include training the autonomousdriving model to learn about a driving area based on information aboutthe gaze direction of the driver and training the autonomous drivingmodel to learn about at least one of a dangerous situation and a levelof danger in driving based on the driving pattern information.

The method may also include: collecting navigation setting informationmay include information on a starting location and an ending location.

The training of the model may also include training the autonomousdriving model to learn about an integrated driving situation from thestarting location to the ending location further based on the navigationsetting information.

The sensor information may also include at least one of surroundingimage information, surrounding voice information, current locationinformation, light detection and ranging (lidar) sensor information,radar sensor information, vehicle speed information, and accelerationinformation.

In accordance with an embodiment, there is provided a non-transitorycomputer-readable storage medium storing instructions that, whenexecuted by a processor, cause the processor to perform the methoddescribed above.

In accordance with another embodiment, there is provided an autonomousdriving apparatus, may include: a driving information collectionprocessor configured to collect vehicle driving information may includeat least one of driver information, sensor information, and navigationsetting information while a vehicle is being driven; and a commandprocessor configured to train an autonomous driving model installed inthe vehicle or autonomously drive the vehicle based on the collecteddriving information.

The autonomous driving apparatus may also include: a command receiverconfigured to receive an autonomous driving command from a driver.

The command processing processor may also include: a model determinationprocessor configured to determine whether the autonomous driving modelis appropriate for a driver and the current driving situation based oneither one or both of the driver information and the navigation settinginformation; a model training processor configured to retrain theautonomous driving model based on the collected driving information upondetermining that the autonomous driving model is inappropriate; and adriving control processor configured to automatically control operationof the vehicle using the collected driving information and theautonomous driving model upon determining that the autonomous drivingmodel is appropriate.

The model determination processor may check whether the autonomousdriving model is trained to learn about the driver based on the driverinformation and determines that the autonomous driving model isinappropriate upon determining that the autonomous driving model isuntrained about the driver.

The model determination processor may check whether the autonomousdriving model is trained to learn about a driving route from a startinglocation to an ending location based on the navigation settinginformation and determines that the autonomous driving model isinappropriate upon determining that the autonomous driving model isuntrained about the driving route.

In accordance with a further embodiment, there is provided an apparatus,may include: a processor configured to collect driver informationcorresponding to behavior information of a driver driving a vehicle,virtually or in real-time, collect sensor information from sensorscapturing either one or both of image and ambient informationsurrounding the vehicle and vehicle operational parameters correspondingto either one or both of a current driving route and current drivingconditions, train an autonomous driving model by combining the driverinformation and the sensor information, and execute the autonomousdriving model upon determining that the trained autonomous driving modelcorresponds to at least one of the driver, the current driving route.

The processor may also include: a driver information collectionprocessor configured to collect the driver information, a sensorinformation collection processor configured to collect the sensorinformation corresponding to either one or both of the current drivingroute and the current driving conditions, a model training processorconfigured to train the autonomous driving model, and a driving controlprocessor configured to execute the autonomous driving model upondetermining that the trained autonomous driving model corresponds to atleast one of the driver, the current driving route.

The apparatus may also include: a navigation information collectionprocessor configured to collect navigation setting information mayinclude information regarding a starting location, a stopover location,and an ending location.

The processor may be further configured to check whether the autonomousdriving model has been trained within a preset period of time to preventa retraining of the autonomous driving model.

Other features and aspects will be apparent from the following detaileddescription, the drawings, and the claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of an apparatus to train an autonomous drivingmodel, according to an embodiment.

FIG. 2 is a block diagram of an apparatus to train an autonomous drivingmodel, according to another embodiment.

FIG. 3 is a diagram for describing acquisition of training data to trainan autonomous driving model.

FIG. 4 is a flowchart showing a method to train an autonomous drivingmodel, according to an embodiment.

FIG. 5 is a flowchart showing a method to train an autonomous drivingmodel, according to another embodiment.

FIG. 6 is a block diagram of an autonomous driving apparatus, accordingto an embodiment.

FIG. 7 is a detailed block diagram of a command processing unitdescribed and illustrated with respect to FIG. 6.

FIG. 8 is a flowchart of an autonomous driving method, according to anembodiment.

FIG. 9 is a detailed flowchart of a command processing of FIG. 8.

Throughout the drawings and the detailed description, unless otherwisedescribed, the same drawing reference numerals should be understood torefer to the same elements, features, and structures. The relative sizeand depiction of these elements may be exaggerated for clarity,illustration, and convenience.

DETAILED DESCRIPTION

The following detailed description is provided to assist the reader ingaining a comprehensive understanding of the methods, apparatuses,and/or systems described herein. However, various changes,modifications, and equivalents of the methods, apparatuses, and/orsystems described herein will be apparent after an understanding of thedisclosure of this application. For example, the sequences of operationsdescribed herein are merely examples, and are not limited to those setforth herein, but may be changed as will be apparent after anunderstanding of the disclosure of this application, with the exceptionof operations necessarily occurring in a certain order. Also,descriptions of features that are known in the art may be omitted forincreased clarity and conciseness.

The features described herein may be embodied in different forms, andare not to be construed as being limited to the examples describedherein. Rather, the examples described herein have been provided merelyto illustrate some of the many possible ways of implementing themethods, apparatuses, and/or systems described herein that will beapparent after an understanding of the disclosure of this application.

As used herein, the term “and/or” includes any one and any combinationof any two or more of the associated listed items.

The terminology used herein is for describing various examples only, andis not to be used to limit the disclosure. The articles “a,” “an,” and“the” are intended to include the plural forms as well, unless thecontext clearly indicates otherwise. The terms “comprises,” “includes,”and “has” specify the presence of stated features, numbers, operations,members, elements, and/or combinations thereof, but do not preclude thepresence or addition of one or more other features, numbers, operations,members, elements, and/or combinations thereof.

Due to manufacturing techniques and/or tolerances, variations of theshapes shown in the drawings may occur. Thus, the examples describedherein are not limited to the specific shapes shown in the drawings, butinclude changes in shape that occur during manufacturing.

The features of the examples described herein may be combined in variousways as will be apparent after an understanding of the disclosure ofthis application. Further, although the examples described herein have avariety of configurations, other configurations are possible as will beapparent after an understanding of the disclosure of this application.

Details of other embodiments are included in the detailed descriptionand drawings. Advantages and features of the described technique, andimplementation methods thereof will be clarified through followingembodiments described with reference to the accompanying drawings. Likereference numerals refer to like elements throughout.

An apparatus and method to train an autonomous driving model, accordingto embodiments, will be described below in detail with reference to theaccompanying drawings. An autonomous driving model training apparatus,according to some embodiments, refers to a structural apparatus to traina model for context awareness and operation determination (hereinafterreferred to as an “autonomous driving model”) to autonomously drive avehicle using a variety of information associated with different drivingsituations. For example, the autonomous driving model may be based on aneural network, but is not limited thereto.

In addition, the autonomous driving model training apparatus may beinstalled in a driving vehicle in order to train and produce anautonomous driving model. Alternatively, the autonomous driving modeltraining apparatus may be implemented in a separate hardware devicewired or wirelessly connected to a vehicle to collect information totrain the autonomous driving model over a network instead of beinginstalled in the vehicle. In this case, the autonomous driving modeltraining apparatus wire or wirelessly receives the collected informationfrom the vehicle to train the autonomous driving model.

FIG. 1 is a block diagram of an autonomous driving model trainingapparatus, according to an embodiment.

Referring to FIG. 1, an autonomous driving model training apparatus 100includes a driver information collection processor 110, a sensorinformation collection processor 120, and a model training processor130.

In order to train an autonomous driving model, the driver informationcollection processor 110 collects a variety of information as trainingdata that is associated with a driver while driving a vehicle, that is,vehicle driving situation, and driver information. In an example, thevehicle driving situation includes a situation in which the driver isactually driving the vehicle on a road and also a situation in which thedriver is virtually driving the vehicle in a virtual situation that isset similar to a real road situation. Another vehicle driving situationmay be defined based on a time of the day or night that the driver isdriving the vehicle, the road conditions, such as paved roads or roughterrain, age of the driver, and ambient/environmental conditions, suchas severe winds, rain, fog, dark, or sunny day. In addition, the vehicledriving situation may refer to a period from the time at which a drivergets in the vehicle to the time at which the driver stops driving andgets off the vehicle.

As an example, the driver information may include driving patterninformation regarding a driving pattern in which the driver responds tosurrounding road conditions the driver is facing while driving. Forexample, the driving pattern information includes information associatedwith a manner in which the driver drives the vehicle such as steeringwheel manipulation information, accelerator pedal manipulationinformation, horn manipulation information, turn-signal lampmanipulation information, emergency button manipulation information, andbrake pedal manipulation information according to various roadconditions such as road curvature, a sudden accident, a rumble striparea, a crosswalk, a traffic signal change, and a traffic congestion.

As another example, the driver information includes driver behavior orbehavioral information regarding behavior the driver casually orhabitually exhibits while driving, irrespective of the surrounding roadconditions. For example, the driver behavior information includesmanipulation information regarding an audio device including a stereosystem, an air conditioning device such as an air conditioner, and amirror device such as a side mirror and a room mirror, navigationmanipulation information, etc.

As another example, the driver information includes driver stateinformation that indicates states of the driver corresponding to varioussurrounding situations or specific behaviors exhibited by the driver.For example, the driver state information includes a gaze direction, aheart rate, an electrocardiogram, and brain waves of a driver, and gripstrength of the driver's hand holding a steering wheel.

The sensor information collection processor 120 collects a variety ofinformation associated with the driving of a vehicle from varioussensors installed in a vehicle. For example, the sensor informationcollection processor 120 collects image information regarding roadconditions, light detection and ranging (lidar) sensor information,radar sensor information, and vehicle operational parameters such asspeed information and acceleration information captured through a cameramodule, a black box, a distance measurement sensor, a lidar sensor, aradar sensor, a speed sensor, and an acceleration sensor.

When the driver information, the vehicle driving situation, and thesensor information are collected to train the autonomous driving model,the model training processor 130 trains the autonomous driving model inconsideration of all of the collected driver information and sensorinformation. In this case, the model training processor 130 createstraining data by combining two or more pieces of the collected thedriver information, the vehicle driving situation, and the sensorinformation and trains the autonomous driving model using the createdtraining data.

As an example, in order to train the autonomous driving model to learnabout an area in which the driver has to be careful while driving, themodel training processor 130 uses gaze direction information of thedriving state information as the training data. In this case, the modeltraining processor 130 creates training data regarding the area in whichthe driver has to be careful by analyzing an image in the gaze directionin which the driver keeps his/her eyes in additional consideration ofsurrounding image information of the sensor information.

As another example, the model training processor 130 processes thedriving pattern information and/or the driver state information tocreate training data regarding dangerous situations and levels of dangerin driving and may train the autonomous driving model using the createdtraining data. For example, in order to create the training data fordangerous situations and considering various levels of danger, the modeltraining processor 130 analyzes how much grip strength a driver has inholding a steering wheel, whether the driver pushed an emergency lampbutton, how much the driver's heart rate is, or how strongly the driverpresses a brake pedal in various driving situations such as a suddenaccident, a curved road, a sudden signal change, and a sudden stop. Inthis case, the model training processor 130 determines the variousdriving situations considering whether the current road is curved orwhether a traffic signal has changed by analyzing the sensorinformation, for example, the surrounding image information.

However, this is merely illustrative, and thus embodiments are notlimited thereto. The model training processor 130 may create thetraining data used to train, considering various situations, needed forthe autonomous driving of a vehicle or various situations associatedwith operations of the vehicle to be controlled by processing variousinformation collected while the vehicle is being driven on a road.

FIG. 2 is a block diagram of an apparatus for training an autonomousdriving model, according to another embodiment.

Referring to FIG. 2, an autonomous driving model training apparatus 200includes a driver information collection processor 210, a sensorinformation collection processor 220, a model training processor 230,and a navigation information collection processor 240. The driverinformation collection processor 210, the sensor information collectionprocessor 220, and the model training processor 230 have been describedin detail with reference to FIG. 1.

As shown, the autonomous driving model training apparatus 200 furtherincludes the navigation information collection processor 240 configuredto collect navigation setting information, for example, informationregarding a starting location, a stopover location, and an endinglocation.

When the navigation setting information is collected by the navigationinformation collection processor 240, the model training processor 230integrally trains the autonomous driving model to learn road driving ona route from the current starting location to an ending location via astopover location by integrally considering a variety of driverinformation collected by the driver information collection processor 210and sensor information collected by the sensor information collectionprocessor 220.

In this case, the model training processor 230 trains an autonomousdriving model for each road driving route according to a startinglocation, a stopover location, and an ending location to establish theautonomous driving model for each road driving route.

FIG. 3 is a diagram to describe acquisition of training data to train anautonomous driving model. As described above in detail, in order totrain an autonomous driving model, according to some embodiments,navigation setting information, a variety of driver informationincluding a gaze direction, brain wave, an electrocardiogram, steeringwheel manipulation information, grip strength information, andturn-signal lamp manipulation information, and other related informationmay be collected in addition to sensor information regarding a camera, alidar, a radar, etc.

In order to train an autonomous driving model that is commonlyapplicable to general driving, various characteristics, includinggender, age, height, weight, etc., are considered when analyzing each ofa plurality of drivers to collect training data as shown in FIG. 3.Also, various driving environments are preset for a plurality ofdrivers, and the training data is collected for each of the drivingenvironments preset for the drivers. For this, the autonomous drivingmodel training apparatuses 100 and 200 manage information setdifferently for each driver, such as driving routes, roads (including,highways, local roads, mountain trails, unpaved roads, etc.), or averagespeed limits or preset information such as a phone conversation orstereo manipulation after a specific location or time.

FIG. 4 is a flowchart showing a method of training an autonomous drivingmodel, according to an embodiment. FIG. 4 shows an example of a trainingmethod performed by the autonomous driving model training apparatus 100of FIG. 1.

At operation 410, the autonomous driving model training apparatus 100collects a variety of driver information associated with a driver whilethe vehicle is in motion or while driving the vehicle. In accord with anembodiment, as described above, the driver information includes thedriving pattern information such as steering wheel manipulationinformation, accelerator pedal manipulation information, hornmanipulation information, turn-signal lamp manipulation information,emergency button manipulation information, and brake pedal manipulationinformation according to various road conditions. In addition, thedriver information includes driver behavior information such as stereodevice manipulation, air conditioning device manipulation, or a phoneconversation the driver casually or habitually does while driving. Inaddition, the driver information may include driver state informationsuch as a gaze direction, a heart rate, an electrocardiogram, brainwaves, grip strength of a hand holding a steering wheel, and additionalinformation.

In addition, at operation 420, the autonomous driving model trainingapparatus 100 collects a variety of sensor information associated with adriving vehicle through various sensors installed in the vehicle. Theinformation is collected in real-time in operations 410 and 420 whilethe vehicle is in motion. In one embodiment, operations 410 and 420 areperformed simultaneously, in real-time. In another embodiment, one ofthe operations 410 and 420 is performed before the other.

Next, when the driver information and the sensor information arecollected, at operation 430, the autonomous driving model trainingapparatus 100 trains an autonomous driving model in consideration of,based on, or by processing all of the collected driver information andsensor information. In this case, the autonomous driving model trainingapparatus 100 creates the training data consistent with variousautonomous driving situations by combining pieces of the collectedvariety of information to train the autonomous driving model to allowaccurate autonomous driving.

For example, in order to train the autonomous driving model to learnabout an area in which the driver has to be careful while driving, theautonomous driving model training apparatus 100 processes gaze directioninformation and sensor information to create the training data. Inaddition, the autonomous driving model training apparatus 100 processesdriving pattern information and/or driver state information to createthe training data regarding a dangerous situation and a level of dangerin driving.

FIG. 5 is a flowchart of a method of training an autonomous drivingmodel, according to another embodiment. FIG. 5 shows an example of atraining method performed by the autonomous driving model trainingapparatus 200 of FIG. 2.

At operation 510, referring to FIG. 5, the autonomous driving modeltraining apparatus 200 may collect information set in a navigationdevice of a vehicle. In an embodiment, the navigation settinginformation includes information regarding routes along which thevehicle will drive, for example, information on a starting location, astopover location, and an ending location.

Next, at operation 520, when the driver starts to drive the vehicle orafter a predefined period of time, the autonomous driving model trainingapparatus 200 collects driver information associated with a driverresponding to various driving situations while the vehicle is in motion.

In addition, when the vehicle is in motion, at operation 530, theautonomous driving model training apparatus 200 collects various typesof sensor information generated by sensors installed in the vehicle.Here, steps 520 and 530 are performed in real time while the vehicle isin motion. In one embodiment, operations 520 and 530 are performedsimultaneously, in real-time. In another embodiment, one of theoperations 520 and 530 is performed before the other.

At operation 540, the autonomous driving model training apparatus 200trains the autonomous driving model in consideration of, based on, or byprocessing all of the collected information.

According to this embodiment, it is possible to collect the navigationsetting information, perform integrated training in consideration of adriving route of a vehicle, and train and establish an autonomousdriving model for each of various driving routes.

FIG. 6 is a block diagram of an autonomous driving apparatus, accordingto an embodiment. An autonomous driving apparatus 600, according to anembodiment, is an apparatus that is installed in a vehicle andconfigured to support an autonomous driving of the vehicle. Theautonomous driving apparatus 600 includes the above-described autonomousdriving model training function.

As shown in FIG. 6, the autonomous driving apparatus 600 includes acommand receiver 610, a driving information collection processor 620,and a command processor 630.

The command receiver 610 receives from a driver various autonomousdriving commands that may be performed by the autonomous drivingapparatus 600. For example, the autonomous driving commands include acommand to switch from a manual driving mode to an autonomous drivingmode or from the autonomous driving mode to the manual driving mode, acommand to switch to an autonomous driving model training mode, or anend command.

When an autonomous driving command is received from the driver, thedriving information collection processor 620 collects vehicle drivinginformation so that the command processor 630 may perform an operationbased on the command. In this case, the vehicle driving information is avariety of information that is created while a vehicle is driving andincludes driver information associated with a particular driver, sensorinformation created from various types of sensors installed in thevehicle, and navigation setting information associated with a drivingroute, which have been created since the time at which driver entered anautonomous driving command. More detailed examples have been describedabove in detail, and thus detailed descriptions thereof will be omitted.

The command processor 630 processes the received autonomous drivingcommand on the basis of the driving information collected by the drivinginformation collection processor 620. That is, when the command isreceived from the driver, the command processor 630 processes thecommand and performs an operation corresponding to the received commandin consideration of all of the collected driving information. Elementsof the command processor 630 will be described in detail with referenceto FIG. 7.

FIG. 7 is a detailed block diagram of the command processor 630 of FIG.6.

Referring to FIG. 7, the command processor 630 may include a modeldetermination processor 631, a model training processor 632, and adriving control processor 633.

The model determination processor 631 processes the received commandfrom the driver and determines an operation to be processed on the basisof a result of the processing. For example, when the autonomous drivingcommand received from the driver is a command to switch a vehicledriving mode to an autonomous driving mode, the model determinationprocessor 631 determines whether an autonomous driving model installedin the vehicle is appropriate before the vehicle driving mode isswitched to the autonomous driving mode. In one embodiment, the modeldetermination processor 631 determines whether an autonomous drivingmodel installed in the vehicle is appropriate by determining whether theautonomous driving model installed in the vehicle corresponds to thedriver to be driving the vehicle, based on the driving conditions, thedriving location, or other information that the driver information andthe sensor information and the navigation setting information include todefine the model determination processor 631.

In addition, when the model determination processor 631 determines thatthe autonomous driving model is appropriate, the model determinationprocessor 631 determines to process the received command to performautonomous driving. On the other hand, when the model determinationprocessor 631 determines that the autonomous driving model is notappropriate, the model determination processor 631 restrains fromexecuting the autonomous driving model.

As an example, through the driver information of the driving informationcollected by the driving information collection processor 620, the modeldetermination processor 631 checks whether the autonomous driving modelis trained in advance or pre-trained with regards to the current driverand determines whether the autonomous driving model is appropriate forthe driver when a result of the checking is that the autonomous drivingmodel is not pre-trained for the current driver.

As another example, the model determination processor 631 checks whetherthe autonomous driving model is pre-trained with regards to the currentdriving route with reference to the navigation setting information ofthe driving information collected by the driving information collectionprocessor 620. When a result of the checking is that the autonomousdriving model is not pre-trained to learn about the current drivingroute, the model determination processor 631 determines that theautonomous driving model is not appropriate for the current drivingsituation.

When it is determined that the autonomous driving model is notappropriate and, thus, the autonomous driving model needs to beretrained, the model determination processor 631 confirms with thedriver about whether to perform the retraining or, regardless, performautonomous driving and determines an operation to be processed based onthe driver's response to the inquiry.

When the driver purchases a new vehicle or desires to drive along a newdriving route, the driver enters a command to the model determinationprocessor 631 to operate the vehicle in a training mode in order toestablish his/her suitable model. In this case, when the commandreceived from the driver is a command to train the autonomous drivingmodel, the model determination processor 631 skips the process ofdetermining whether the autonomous driving model is appropriate for thecurrent driver or driving route. However, embodiments of the presentinvention are not limited thereto. In some cases, the modeldetermination processor 631 checks whether the current autonomousdriving model has been recently trained even when the driver enters thetraining command. When the training has been recently performed, withina preset period of time, such as an hour, a day, a week, or other timeperiod, the model determination processor 631 determines that theretraining need not be performed. In this case, the model determinationprocessor 631 provides the driver with information indicating thatadditional training is not needed or request the driver to confirmwhether to perform the retraining once again.

When the model determination processor 631 determines that theautonomous driving model needs to be trained, the model trainingprocessor 632 creates training data based on all of the driverinformation, the sensor information, and the navigation settinginformation collected by the driving information collection processor620 and trains the autonomous driving model using the created trainingdata. In this case, the model training processor 632 applies theabove-described autonomous driving model training technique to train theautonomous driving model.

In order to perform the training sufficiently using the drivinginformation, the model training processor 632 trains the autonomousdriving model at a predetermined time after the command is entered fromthe driver. In one embodiment, the predetermine time at which thetraining is performed may be preset by a manufacturer when theautonomous driving apparatus 600 is installed in the vehicle.

When the model determination processor 631 determines that the commandof the driver is to perform an autonomous driving, the driving controlprocessor 633 switches a vehicle driving mode from a manual mode inwhich the driver manipulate a vehicle to an autonomous driving mode andautomatically controls an operation of the vehicle using the drivingmode collected by the driving information collection processor 620 andthe autonomous driving mode.

The driving control processor 633 may automatically control theoperation of the vehicle to perform an operation corresponding to thedriver's habit or driving pattern according to the current drivingsituation by entering the navigation setting information and the sensorinformation collected in real-time into the autonomous driving model.

According to an embodiment, even when the vehicle operates in anautonomous driving mode, it is possible to control the operation of thevehicle similarly as in a situation in which the driver is activelydriving the vehicle because the autonomous driving model is trained inconsideration of the driver's habit or driving pattern.

FIG. 8 is a flowchart of an autonomous driving method, according to anembodiment.

Referring to FIG. 8, at operation 810, in response to a driver enteringan autonomous driving command while driving a vehicle, the autonomousdriving apparatus 600 receives the command from the driver. In anembodiment, the autonomous driving command includes a command to switchto an autonomous driving mode or to a manual driving mode, a command forswitching to an autonomous driving model training mode, or an endcommand.

At operation 820, when the autonomous driving command is received fromthe driver, the autonomous driving apparatus 600 collects drivinginformation that is used to process the received command. In anembodiment, the vehicle driving information is a variety of informationthat is created while a vehicle is driving and includes the driverinformation associated with a driver, the sensor information createdfrom various types of sensors installed in the vehicle, and thenavigation setting information associated with a driving route, whichhave been created since or prior to the time at which driver entered anautonomous driving command.

At operation 930, the autonomous driving apparatus 600 processes acommand the driver requests using the received driving information.

FIG. 9 is a detailed flowchart of the command processing operation 830of FIG. 8. Operation 830 will be described in detail below withreference to FIGS. 8 and 9.

At operation 810, when the autonomous driving command is received fromthe driver, the autonomous driving apparatus 600 interprets the receivedcommand to determine, at operation 831, whether to process the receivedcommand to operate the vehicle in an autonomous driving mode, in whichthe driving of a vehicle is automatically controlled, or to operate thevehicle in a training mode, in which an autonomous driving modelinstalled for autonomous driving is retrained. At operation 831, theautonomous driving apparatus 600 determines whether the currentautonomous driving model applied to the vehicle is appropriate.

For example, when the autonomous driving command received from thedriver is to operate the vehicle in the training mode, the autonomousdriving apparatus 600 determines to operate the vehicle in the trainingmode. In this case, when the current autonomous driving model is onethat has been recently trained, the autonomous driving apparatus 600inquires the driver of whether to perform the training and performs thetraining on the basis of the driver's response to the inquiry.

On the other hand, when the autonomous driving command is to operate thevehicle in the autonomous driving mode, the autonomous driving apparatus600 determines that an autonomous driving model to be applied isappropriate based on the driver information or the navigation settinginformation before operating the vehicle in the autonomous driving mode.As an example, when a driver purchases a new vehicle or when a driver ischanged because an owner of the current vehicle has changed, that is,when an autonomous driving model applied to the current vehicle is nottrained for a particular driver, the autonomous driving apparatus 600determines that the autonomous driving model is not appropriate for ordoes not correspond to the driver driving the vehicle, or is not trainedfor a particular driving route or is not trained for current drivingconditions. As another example, when the current autonomous drivingmodel is not trained for a new driving route, the autonomous drivingapparatus 600 determines that the autonomous driving model is notappropriate. When the autonomous driving apparatus 600 determines thatthe current autonomous driving model is not appropriate, the autonomousdriving apparatus 600 operates the vehicle in the training mode. In thiscase, the autonomous driving apparatus 600 informs the driver that thevehicle will operate in the training mode and may finally determine anoperation to be processed on the basis of the driver's response.

At operation 831, when the autonomous driving apparatus 600 determinesthat the autonomous driving model currently applied is not appropriate,at operation 832, the autonomous driving apparatus 600 trains theautonomous driving model using the collected driving information. Inthis case, the autonomous driving apparatus 600 uses the above-describedautonomous driving model training technique and trains the autonomousdriving model after driving information is collected for a predeterminedtime in order to perform the training using a sufficient amount ofdriving information.

At operation 833, in response to the training of the autonomous drivingmodel being completed, the autonomous driving model training apparatus200 determines whether to operate the vehicle in the autonomous drivingmode. In this example, the autonomous driving apparatus 600 informs ornotifies the driver that the training has been completed and inquireswhether to keep performing automatic control of the driving of thevehicle or operate the vehicle in the autonomous driving mode accordingto a predetermined rule.

Next, in response to the autonomous driving apparatus 600 determining inoperation 831 that the autonomous driving model is appropriate for ordoes correspond to the driver driving the vehicle, or is trained for theparticular driving route or is trained for the current drivingconditions or determines in operation 833 that the vehicle is operatingin the autonomous driving mode, at operation 834, the autonomous drivingapparatus 600 automatically controls an operation of the vehicle usingthe autonomous driving mode and the driving information collected, inreal-time, in operation 820.

The driver information collection processor 110 and 210, the sensorinformation collection processor 120 and 220, the model trainingprocessor 130 and 230, the navigation information collection processor240, the command receiver 610, the driving information collectionprocessor 620, the command processor 630, model determination processor631, the model training processor 632, and the driving control processor633 in FIGS. 1-3 and 6-7 that perform the operations described in thisapplication are implemented by hardware components configured to performthe operations described in this application that are performed by thehardware components. Examples of hardware components that may be used toperform the operations described in this application where appropriateinclude controllers, sensors, generators, drivers, memories,comparators, arithmetic logic units, adders, subtractors, multipliers,dividers, integrators, and any other electronic components configured toperform the operations described in this application. In other examples,one or more of the hardware components that perform the operationsdescribed in this application are implemented by computing hardware, forexample, by one or more processors or computers. A processor or computermay be implemented by one or more processing elements, such as an arrayof logic gates, a controller and an arithmetic logic unit, a digitalsignal processor, a microcomputer, a programmable logic controller, afield-programmable gate array, a programmable logic array, amicroprocessor, or any other device or combination of devices that isconfigured to respond to and execute instructions in a defined manner toachieve a desired result. In one example, a processor or computerincludes, or is connected to, one or more memories storing instructionsor software that are executed by the processor or computer. Hardwarecomponents implemented by a processor or computer may executeinstructions or software, such as an operating system (OS) and one ormore software applications that run on the OS, to perform the operationsdescribed in this application. The hardware components may also access,manipulate, process, create, and store data in response to execution ofthe instructions or software. For simplicity, the singular term“processor” or “computer” may be used in the description of the examplesdescribed in this application, but in other examples multiple processorsor computers may be used, or a processor or computer may includemultiple processing elements, or multiple types of processing elements,or both. For example, a single hardware component or two or morehardware components may be implemented by a single processor, or two ormore processors, or a processor and a controller. One or more hardwarecomponents may be implemented by one or more processors, or a processorand a controller, and one or more other hardware components may beimplemented by one or more other processors, or another processor andanother controller. One or more processors, or a processor and acontroller, may implement a single hardware component, or two or morehardware components. A hardware component may have any one or more ofdifferent processing configurations, examples of which include a singleprocessor, independent processors, parallel processors,single-instruction single-data (SISD) multiprocessing,single-instruction multiple-data (SIMD) multiprocessing,multiple-instruction single-data (MISD) multiprocessing, andmultiple-instruction multiple-data (MIMD) multiprocessing.

The methods illustrated in FIGS. 4-5 and 8-9 that perform the operationsdescribed in this application are performed by computing hardware, forexample, by one or more processors or computers, implemented asdescribed above executing instructions or software to perform theoperations described in this application that are performed by themethods. For example, a single operation or two or more operations maybe performed by a single processor, or two or more processors, or aprocessor and a controller. One or more operations may be performed byone or more processors, or a processor and a controller, and one or moreother operations may be performed by one or more other processors, oranother processor and another controller. One or more processors, or aprocessor and a controller, may perform a single operation, or two ormore operations.

Instructions or software to control computing hardware, for example, oneor more processors or computers, to implement the hardware componentsand perform the methods as described above may be written as computerprograms, code segments, instructions or any combination thereof, forindividually or collectively instructing or configuring the one or moreprocessors or computers to operate as a machine or special-purposecomputer to perform the operations that are performed by the hardwarecomponents and the methods as described above. In one example, theinstructions or software include machine code that is directly executedby the one or more processors or computers, such as machine codeproduced by a compiler. In another example, the instructions or softwareincludes higher-level code that is executed by the one or moreprocessors or computer using an interpreter. The instructions orsoftware may be written using any programming language based on theblock diagrams and the flow charts illustrated in the drawings and thecorresponding descriptions in the specification, which disclosealgorithms for performing the operations that are performed by thehardware components and the methods as described above.

The instructions or software to control computing hardware, for example,one or more processors or computers, to implement the hardwarecomponents and perform the methods as described above, and anyassociated data, data files, and data structures, may be recorded,stored, or fixed in or on one or more non-transitory computer-readablestorage media. Examples of a non-transitory computer-readable storagemedium include read-only memory (ROM), random-access memory (RAM), flashmemory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs,DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, magnetictapes, floppy disks, magneto-optical data storage devices, optical datastorage devices, hard disks, solid-state disks, and any other devicethat is configured to store the instructions or software and anyassociated data, data files, and data structures in a non-transitorymanner and provide the instructions or software and any associated data,data files, and data structures to one or more processors or computersso that the one or more processors or computers can execute theinstructions. In one example, the instructions or software and anyassociated data, data files, and data structures are distributed overnetwork-coupled computer systems so that the instructions and softwareand any associated data, data files, and data structures are stored,accessed, and executed in a distributed fashion by the one or moreprocessors or computers.

While this disclosure includes specific examples, it will be apparentafter an understanding of the disclosure of this application thatvarious changes in form and details may be made in these exampleswithout departing from the spirit and scope of the claims and theirequivalents. The examples described herein are to be considered in adescriptive sense only, and not for purposes of limitation. Descriptionsof features or aspects in each example are to be considered as beingapplicable to similar features or aspects in other examples. Suitableresults may be achieved if the described techniques are performed in adifferent order, and/or if components in a described system,architecture, device, or circuit are combined in a different manner,and/or replaced or supplemented by other components or theirequivalents. Therefore, the scope of the disclosure is defined not bythe detailed description, but by the claims and their equivalents, andall variations within the scope of the claims and their equivalents areto be construed as being included in the disclosure.

What is claimed is:
 1. An autonomous driving apparatus comprising: adriving information collection processor configured to collect drivinginformation of a vehicle; and a model training processor configured totrain an autonomous driving model based on the driving information. 2.The autonomous driving apparatus of claim 1, wherein the drivinginformation comprises at least one of a driver information, a sensorinformation from a sensor installed in the vehicle, and a navigationsetting information.
 3. The autonomous driving apparatus of claim 2,wherein the driver information comprises at least one of a driver stateinformation, a driver behavior information, and a driving patterninformation.
 4. The autonomous driving apparatus of claim 3, wherein:the driver state information comprises at least one piece of informationabout a gaze direction, a heart rate, an electrocardiogram, brain waves,and grip strength of a hand holding a steering wheel of a driver; thedriver behavior information comprises at least one piece of manipulationinformation about an audio device, an air conditioner, a mirror device,and a navigation device installed in the vehicle; and the drivingpattern information comprises at least one of steering wheelmanipulation information, accelerator pedal manipulation information,horn manipulation information, turn-signal manipulation information,emergency button manipulation information, and brake pedal manipulationinformation.
 5. The autonomous driving apparatus of claim 4, wherein themodel training processor is configured to: train the autonomous drivingmodel to learn about an area in which the driver has to be careful whiledriving based on the gaze direction of the driver; and train theautonomous driving model to learn about at least one of dangeroussituations and levels of danger in driving based on the driving patterninformation.
 6. The autonomous driving apparatus of claim 1, wherein themodel training processor trains the autonomous driving model based onthe driving information after collection of the driving informationstarts and a predefined period of time has elapsed.
 7. The autonomousdriving apparatus of claim 2, wherein the model training processortrains the autonomous driving model to learn about an integrated drivingsituation from a starting location to an ending location based on thenavigation setting information.
 8. The autonomous driving apparatus ofclaim 2, wherein the sensor information comprises at least one ofsurrounding image information, surrounding voice information, currentlocation information, light detection and ranging (LiDAR) sensorinformation, radar sensor information, vehicle speed information, andacceleration information.
 9. The autonomous driving apparatus of claim1, further comprising a model determination processor configured todetermine whether the autonomous driving model is appropriate for adriver and a current driving situation based on at least one of a driverinformation and a navigation setting information.
 10. The autonomousdriving apparatus of claim 9, further comprising a driving controllerconfigured to automatically control an operation of the vehicle when aresult of the model determination processor is the autonomous drivingmodel is appropriate for the driver and the current driving situation.11. The autonomous driving apparatus of claim 9, wherein the modeldetermination processor is configured to: check whether the autonomousdriving model is trained for the driver based on the driver information;and when a result of the check is that the autonomous driving model isnot trained for the driver, determine that the autonomous driving modelis not appropriate.
 12. The autonomous driving apparatus of claim 9,wherein the model determination processor is configured to: checkwhether the autonomous driving model is trained for a driving route froma starting location to an ending location based on the navigationsetting information; and when a result of the check is that theautonomous driving model is not trained for the driving route, determinethat the autonomous driving model is not appropriate.
 13. An autonomousdriving method comprising: collecting driving information of a vehicle;and training, an autonomous driving model based on the drivinginformation.
 14. The autonomous driving method of claim 13, wherein thedriving information comprises at least one of a driver information, asensor information from a sensor installed in the vehicle, and anavigation setting information.
 15. The autonomous driving method ofclaim 14, wherein: the driver information comprises at least one of adriver state information, a driver behavior information, and a drivingpattern information; the driver state information comprises at least onepiece of information about a gaze direction, a heart rate, anelectrocardiogram, brain waves, and grip strength of a hand holding asteering wheel of a driver; the driver behavior information comprises atleast one piece of manipulation information about an audio device, anair conditioner, a mirror device, and a navigation device installed inthe vehicle; and the driving pattern information comprises at least oneof steering wheel manipulation information, accelerator pedalmanipulation information, horn manipulation information, turn-signalmanipulation information, emergency button manipulation information, andbrake pedal manipulation information.
 16. The autonomous driving methodof claim 15, wherein the training of the autonomous driving modelcomprises: training the autonomous driving model to learn about an areain which the driver has to be careful while driving based on the gazedirection of the driver; and training the autonomous driving model tolearn about at least one of dangerous situations and levels of danger indriving based on the driving pattern information.
 17. The autonomousdriving method of claim 13, wherein the training of the autonomousdriving model comprises training, after the collecting of the drivinginformation starts and a predefined period of time has elapsed, theautonomous driving model based on the driving information.
 18. Theautonomous driving method of claim 14, wherein the training of theautonomous driving model comprises training the autonomous driving modelto learn about an integrated driving situation from a starting locationto an ending location based on the navigation setting information. 19.The autonomous driving method of claim 13, further comprisingdetermining whether the autonomous driving model is appropriate for adriver and a current driving situation based on at least one of driverinformation and navigation setting information.
 20. The autonomousdriving method of claim 19, further comprising automaticallycontrolling, when a result of determining the autonomous driving modelis appropriate for the driver and the current driving situation, anoperation of the vehicle.